Deep Learning Speech Engine/Voice Assistant

I would like to build a Speech engine/Voice Assistant using deeplearning4j or tensorflow using DeepLearning or Machine Learning models. Few Use cases are like below - Use Case 1 Agent: what do you want to book? User: I want to book a taxi Agent: please provide the start and end location, time etc User: From 1 Zero street, Belconnen to 2, Three street, Woden at 3pm Agent: Agent opens Uber and books a taxi and confirms the user Use Case 2 User: Tom, can you book a room for 2 adult and 1 children on 1 Nov 2017? Agent: takes the instruction and opens AirBNB, searches as per search criteria and recommends the best match hotels from the search results and starts showing them. User: specifies the chosen one Agent: Proceeds with the booking