Future Internet, described as an “Internet of Things,” is planned to be a global network of connected items, each with a unique address, based on industry-standard protocol. It is an important growing technology for environmental monitoring and future enterprises. IoT could be described as linking commonplace objects to the Internet, such as smart phones, actuators and sensors to enable new communication forms between objects themselves as well as between objects and people. Internet of Things (IoT) and wireless sensor networks (WSN) can be used to perform smart home technologies. This research presented Ad hoc routing protocols in IoT -based WSN in smart home system using Riverbed Modeler simulation platform. The simulation of WSN based on Mesh topology-ZigBee (IEEE 802.15.4) standard. Different applications such as Data Access, File transfer, Peer - to peer File sharing, Voice and Video, Mobile Messaging were applied in different number of scenarios of IoT based Wireless Sensor Network with three routing protocols (AODV, OLSR and GRP hybrid routing protocol) were taken in this study. In different modeled scenarios of this study, the sensing nodes (sensors) sense the environmental condition and send the collected data to the WSN controller which it is represented by ZigBee coordinator. The controller sent the sensor's data to the WiFi which act a gateway, so that this data could be monitored and controlled by the user via the Internet. The research outcomes showed that ad hoc routing protocol played an important role to improve the network's performance in terms of QoS parameters (delay, throughput and data dropped) due to the network deficiency which occurs because of interference between WSN and WiFi since they utilize free frequency band 2.4GHz. in this study, AODV investigated better improvement on the throughput and delay network performance with acceptable improvement in data dropped.