We introduce a minimal Agent Based Model which includes the following elements (first considered by Lux and Marchesi): - Fundamentalists (F: stabilizing tendency) - Chartists (C: destabilizing tendency) - Herding effect (tendency to follow the others) - Price behavior (analysis of the price time series according to F or C criteria) The novelty of our model is a substantial simplification and corresponding reduction of the number of parameters. This leads to a detailed understanding of the origin of the Stylized Facts (SF) like the fat tails and volatility clustering. The SF are shown to correspond to finite size effects (with respect to time and to the number of agents N) which, however, can be active at different time scales. This implies that universality cannot be expected in describing these properties in terms of effective critical exponents. A basic question is then the self-organization: why the system chooses to stay in this narrow range of parameters corresponding also to a finite value of N? We show that the introduction of a threshold in the agents’ action (small price fluctuations lead to no action) triggers the self-organization towards the quasi-critical state and to a finite average value of N (which depends on the other parameters). Non stationarity in the number of active agents and in their action plays a fundamental role. The interpretation of N as the number of effective independent agents is non trivial and deserves further studies. The model can be easily generalized to more realistic variants in a systematic way.