Precision agriculture leverages remote sensing and AI models to monitor crop conditions, but traditional RGB and panchromatic sensors are limited by their fixed optics and reduced spectral selectivity. These limitations hinder the accurate classification of materials in complex agricultural scenes. This paper proposes a compact and passive imaging system based on a diffractive optical element (DOE), co-optimized with a spatial classification model via an end-to-end (E2E) learning strategy. The DOE is parameterized through Zernike polynomials and embedded within a differentiable light propagation model based on angular spectrum theory. The classification network is trained using a single timestamp from the CALCROP21 dataset, ensuring realistic supervision while reducing temporal complexity. Two sensor configurations are evaluated, RGB and panchromatic, each compared under a traditional imaging system using a lens and an imaging system based on a DOE. The results show consistent gains in spatial classification accuracy for both configurations, with improvements of up to 19.7% in test accuracy when comparing the 1-band model with a trainable DOE against its non-trainable counterpart. Additional ablation studies highlight the impact of DOE initialization and demonstrate robustness to additive noise, confirming the effectiveness of task-specific DOE design for embedded agricultural vision systems.