Dry electrodes provide the possibility of moving EEG usage from the research and clinical environment to real life applications. Having a framework for evaluating the performance of dry electrodes would facilitate this process and help EEG system developers to test their designs. This paper describes an evaluation method for dry electrode EEG recordings. The framework includes a setup for synchronous recordings with a parallel dry and gel electrode montage procedure. Several protocols are implemented to evaluate both the time and frequency content of the signal and to compute the setup and settling time. Signal quality was evaluated using signal correlations, SNR and P300 component characteristics. The preliminary data analysis and results show that a comparison between gel and dry electrodes is possible but improvements need to be made to the current evaluation framework.