With the significant increase in integration of renewable energy generation into the electric grid, market-based transactive exchanges between energy producers and prosumers will become more common. Transactive energy systems (TES) employ economic and control mechanisms to dynamically balance the demand and supply across the electrical grid. Emerging transactive control mechanism depends on a large number of distributed edge-computing and Internet of Things (IoT) devices making autonomous/semi-autonomous decisions on energy production, energy consumption and demand response. However, the electric grid cyber assets and the IoT devices are increasingly vulnerable to attack. TES will likely have similar vulnerabilities and cyber attacks specially with financial interest motives of stakeholders, which could affect the operation of the power grid. Therefore, new analytical methods are needed to continuously monitor these systems operations and detect malicious activity. In this research work, various components of transactive energy systems are modeled and simulated in detail. Various cyber attack models are developed based on identified vulnerabilities of TES. A deep learning approach called deep stacked autoencoder (SAE) and MAE based technique are utilized to detect possible anomalies in market and physical system measurements. The proposed technique is validated for satisfactory performance to detect anomalies and trigger further investigation for root cause analysis.