Thanks for contacting us, and sorry for any confusion! When using the WordPress plug-in, your WordPress installation sends the document records to your Swiftype engine directly. Modifying the robots.txt affects only crawl based engines.
What you can do however, is exclude specific WP post types from being synced to your engine, as well as filter by post type, or category. We’ve some additional documentation on how to add these customizations in our help center:
You could also create a crawler based engine instead, which will be able to pick up your robots.txt and custom meta tag settings. That method will require some additional customizations of your WP theme to replace the search, but we’ve a detail of the concept here: https://swiftype.com/documentation/tutorials/add_search_to_any_cms
Please feel free to let me know if you have any additional questions, or if we can help with anything else.