About Me
Ajay Tanwani
Research Lead,
Bosch, USA
Research Interests
- I believe in harnessing AI to transform the way we understand and interact with the world.
- To this end, I develop and deploy machine learning techniques to tackle real-world challenges, spanning perception, natural language, controls, and AI-based reasoning.
- My expertise lies in large-scale representation learning, adapting and deploying deep models via Cloud/Fog for robotics and healthcare applications.
Recent News
Our paper, Large Language Models Encode Clinical Knowledge, is now on arxiv and Nature.
Two papers in Int. J. of Robotics Research (IJRR), and one in Autonomous Robots, 2021.
DIRL: Domain Invariant Representation Learning for Sim-to-Real Transfer is accepted at CoRL 2020. Selected for plenary talk and Best Systems Paper Finalist Award. [pdf]
Motion2Vec featured online by various media partners across the globe including VentureBeat, Engadget, Yahoo, Wall Street Journal, The Next Web and Berkeley Science Magazine.
RILaaS is accepted at IEEE Robotics and Automation Letters and IROS 2020. See the video presentation here
Two papers on robot grasping at CASE 2020.
VisuoSpatial foresight for fabric manipulation is accepted at Robotics, Science and Systems 2020.
Two papers on robot learning have been accepted at ISRR 2019.
We are organizing a workshop on Cloud and Fog Robotics (CAFR) in collaboration with Stanford University at RSS 2019.
I gave an invited talk on Generative Models for Robot Learning at UC Merced.
Two papers on imitation learning have been accepted at WAFR 2018.
Small Variance Asymptotics for Non-Parametric Online Robot Learning is accepted at Int. J. of Robotics Research (IJRR) 2019.
I joined the program/reviewer committee of AAAI 2019, RAL, ICRA 2019, IROS 2019, UAI 2019, ICML 2019, WAFR 2018, NIPS 2018-19, RSS 2018, AABI (NIPS) 2017.
Our NSF Proposal on Fog Robotics has been accepted, in collaboration with SwarmLab and RiseLab. [3 million USD for 3 years]
New York Times article on How Robot Hands are Evolving to Do What Ours Can by Mae Rayan, Cade Metz, and Rumsey Taylor. Our work on the gripper, the picker and the bed-maker is featured against the OpenAI dactyl and anthropomorphic UW hand.
We won the initial round of HSR Home Chores Challenge 2018 [video], and are runners-up in TRI University Challenge 2018 and Siemens FutureMaker Challenge 2018.
Dexterous Underwater Manipulation from Distant Onshore Locations is accepted for publication in IEEE Robotics and Automation Magazine.
We presented two papers on imitation learning in Imitation and Causality Workshop, RSS 2018.
Dex-Net as a Service (DNaaS) is now available online for computing and visualizing robust grasps.
I moved to UC Berkeley for postdoctoral research.
I defended my PhD thesis at EPFL. Thanks to all the jury members !
I finished my internship at Google X in Mountain View, California, 2017.
Publications
- [New] K. Singhal*, S. Azizi*, T. Tu, S. Mahdavi, J. Wei, H. Chung, N. Scales, A. K. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, N. Scharli, A. Chowdhery, P. Mansfield, B. Aguera y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Rajkomar, J. Barral, C. Semturs, A. Karthikesalingam, V. Natarajan. Large Language Models Encode Clinical Knowledge, Nature, 2023.
[pdf] [media] [nature]
- [New] Ajay Kumar Tanwani, Joelle Barral, Daniel Freedman. RepsNet: Combining Vision with Language for Automated Medical Reports, Medical Image Computing and Computer Assisted Intervention, MICCAI, 2022.
[pdf] [bibtex] [website] [demo] [poster] [talk]
- [Best Systems Paper Award Finalist at CoRL 2020] Ajay Kumar Tanwani. DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer, 4th Conference on Robot Learning (CoRL), MIT, 2020. [Plenary Talk]
[pdf] [bibtex] [website] [code]
- [New] Ajay Kumar Tanwani, Raghav Anand, Joseph E. Gonzalez, Ken Goldberg. RILaaS: Robot Inference and Learning as a Service, IEEE Robotics and Automation Letters (RA-L), presented in IROS, 2020
[pdf] [bibtex] [website] [video presentation] [beta version]
- [New] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, K. Yamane, S. Iba, J. F. Canny, K. Goldberg. Deep Imitation Learning of Sequential Fabric Smoothing from an Algorithmic Supervisor, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020
[pdf] [bibtex]
- [New] Jingyi Song, Ajay Kumar Tanwani, J. Ichnowski, M. Danielczuk, K. Sanders, J. Chui, J. A. Ojea, Ken Goldberg. Robust Task-Based Grasping As a Service, Conference on Automation Science and Engineering (CASE), 2020
[pdf] [bibtex] [website]
- [New] Kate Sanders, Michael Danielczuk, Jeffrey Mahler, Ajay Kumar Tanwani, Ken Goldberg. Non-Markov Policies to Reduce Sequential Failures in Robot Bin Picking, Conference on Automation Science and Engineering (CASE), 2020
[pdf] [bibtex]
- [New] Ajay Kumar Tanwani, Pierre Sermanet, Andy Yan, Raghav Anand, Mariano Phielipp, Ken Goldberg. Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos, IEEE International Conference on Robotics and Automation (ICRA), 2020
[pdf] [bibtex] [arxiv] [website] [video] [presentation]
Media: [VentureBeat] [Engadget] [Yahoo] [Wall Street Journal] [The Next Web] [Berkeley Science]
- [New] Ryan Hoque*, Daniel Seita*, Ashwin Balakrishna, Aditya Ganapathi, Ajay Kumar Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, Ken Goldberg. VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation, Robotics, Science and Systems (RSS), 2020
[pdf] [bibtex]
- Daniel Seita*, Nawid Jamali*, Michael Laskey*, Ajay Kumar Tanwani, Ron Berenstein, Prakash Baskaran, Soshi Iba, John Canny, Ken Goldberg. Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making, International Symposium on Robotics Research (ISRR), Hanoi, Vietnam, October 2019
[pdf] [bibtex] [arxiv] [video]
- Nan Tian, Ajay Kumar Tanwani, Ken Goldberg and Somayeh Sojoudi. Mitigating Network Latency in Cloud-Based Teleoperation using Motion Segmentation and Synthesis, International Symposium on Robotics Research (ISRR), Hanoi, Vietnam, October 2019
[pdf] [bibtex] [video]
- Benno Staub, Ajay Kumar Tanwani, Jeffrey Mahler, Michel Breyer, Michael Laskey, Yutaka Takaoka, Max Bajracharya, Roland Siegwart, Ken Goldberg. Dex-Net MM: Deep Grasping for Surface Decluttering with a Low-Precision Mobile Manipulator, IEEE International Conference on Automation Science and Engineering (CASE), 2019
[pdf] [bibtex]
- Ajay Kumar Tanwani, Nitesh Mor, John Kubiatowicz, Joseph E. Gonzalez, Ken Goldberg. A Fog Robotics Approach to Deep Robot Learning: Application to Object Recognition and Grasp Planning in Surface Decluttering, IEEE International Conference on Robotics and Automation (ICRA), 2019
[pdf] [bibtex] [arxiv] [website] [video]
- Dezhen Song, Ajay Kumar Tanwani, Ken Goldberg. Networked-, Cloud- and Fog-Robotics, Robotics goes MOOC, Springer Nature MOOCs, Bruno Siciliano (Editor), Springer. 2019
[pdf] [bibtex]
- Ajay Kumar Tanwani, Sylvain Calinon. Small Variance Asymptotics for Non-Parametric Online Robot Learning. International Journal of Robotics and Research, 2019
[pdf] [bibtex] [arxiv] [video1] [video2]
- Ajay Kumar Tanwani, Jonathan Lee, Brijen Thananjeyan, Michael Laskey, Sanjay Krishnan, Roy Fox, Ken Goldberg, Sylvain Calinon. Generalizing Robot Imitation Learning with Invariant Hidden Semi-Markov Models. Algorithmic Foundations on Robotics WAFR, 2018
[pdf] [bibtex]
- Jonathan Lee, Michael Laskey, Ajay Kumar Tanwani, Anil Aswani, Ken Goldberg. A Dynamic Regret Analysis and Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning. Algorithmic Foundations on Robotics WAFR, 2018
[pdf] [bibtex]
- Pusong Li, Bill DeRose, Jeffrey Mahler, Juan Aparicio Ojea, Ajay Kumar Tanwani, Ken Goldberg. Dex-Net as a Service (DNaaS): A Cloud-Based Robust Robot Grasp Planning System. IEEE International Conference on Automation Science and Engineering (CASE), 2018
[pdf] [bibtex] [website] [media]
- A. Birk, T. Fromm, C.A. Mueller, T. Luczynski, A.G. Chavez, D. Koehntopp, A. Kupcsik, S. Calinon, A.K. Tanwani, G. Antonelli, P. di Lillo, E. Simetti, G. Casalino, G. Indiveri, L. Ostuni, A. Turetta, A. Caffaz, P. Weiss, T. Gobert, B. Chemisky, J. Gancet, T. Siedel, S. Govindaraj, X. Martinez, and P. Letier. Dexterous Underwater Manipulation from Distant Onshore Locations. IEEE Robotics and Automation Magazine, 2018
[pdf] [bibtex]
- Ajay Kumar Tanwani, Jonathon Lee, Michael Laskey, Sanjay Krishnan, Roy Fox, Ken Goldberg. Robot Learning with Invariant Hidden Semi-Markov Models. Imitation and Causality Workshop in Robotics, Science and Systems, 2018
[pdf] [bibtex] [poster]
- Jonathon Lee, Michael Laskey, Ajay Kumar Tanwani, Ken Goldberg. Stability Analysis of On-Policy Imitation Learning Algorithms Using Dynamic Regret . Imitation and Causality Workshop in Robotics, Science and Systems, 2018
[pdf] [bibtex] [poster]
- [PhD Thesis] Ajay Kumar Tanwani. Generative Models for Learning Robot Manipulation Skills from Humans . PhD Thesis, Ecole Polytechnique Federale de Lausanne, EPFL, 2018
[pdf] [multimedia] [bibtex]
- Ajay Kumar Tanwani, Sylvain Calinon. A Generative Model for Intention Recognition and Manipulation Assistance in Teleoperation . IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017
[pdf] [bibtex] [video1] [video2] [media]
- Ajay Kumar Tanwani, Sylvain Calinon. Small Variance Asymptotics for Non-Parametric Online Robot Learning. arXiv:1610.02468, 2016
[pdf] [bibtex] [video1] [video2]
- Ajay Kumar Tanwani, Sylvain Calinon. Online Inference in Bayesian Non-Parametric Mixture Models under Small Variance Asymptotics . NIPS workshop on Advances in Approximate Bayesian Inference, 2016
[pdf] [bibtex] [poster]
- J. Gancet, P. Weiss, G. Antonelli, MF. Pfingsthorn, S. Calinon, A. Turetta, C. Walen, D. Urbina, S. Govindaraj, P. Letier, X. Martinez, J. Salini, B. Chemisky, G. Indiveri, G. Casalino, P. Di Lillo, E. Simetti, D. De Palma, A. Birk, A. K. Tanwani, I. Havoutis, A. Caffaz, L. Guilpain. Dexterous Undersea Interventions with Far Distance Onshore Supervision: the DexROV Project . IFAC Conference on Control Applications in Marine Systems (CAMS), 2016
[pdf] [bibtex] [video]
- Ioannis Havoutis, Ajay Kumar Tanwani and Sylvain Calinon, Online Incremental Learning of Manipulation Tasks for Semi-Autonomous Teleoperation . IEEE/RSJ IROS Workshop on Closed-loop Grasping and Manipulation, 2016
[pdf] [bibtex] [video]
- [Best Manipulation Paper Award Finalist at ICRA 2016] Ajay Kumar Tanwani, Sylvain Calinon. Learning Robot Manipulation Tasks with Task-Parameterized Semi-Tied Hidden Semi-Markov Model. IEEE Robotics and Automation Letters (RA-L), vol. 1 (1), pp. 235-242, 2016. presented in International Conference on Robotics and Automation (ICRA) 2016
[pdf] [bibtex] [video] [codes] [slides] [media1][media2]
- Ajay Kumar Tanwani. Learning Robot Manipulation Tasks with Semi-Tied Gaussian Mixture Models. Swiss Machine Learning Day, EPFL, 2015
[link] [slides]
- J. Gancet, D. Urbina, P. Letier, M. Ilzokvitz, P. Weiss, F. Gauch, G. Antonelli, G. Indiveri, G. Casalino, A. Birk, MF. Pfingsthorn, S. Calinon, A. K. Tanwani, A. Turetta, C. Walen, L. Guilpain. DexROV: Dexterous Undersea Inspection and Maintenance in Presence of Communication Latencies. IFAC Workshop on Navigation, Guidance and Control of Underwater Vehicles (NGCUV), 2015
[pdf] [bibtex] [video]
- Ajay Kumar Tanwani, José del R. Millán, Aude Billard. Rewards-Driven Control of Robot Arm by Decoding EEG Signals. IEEE Engineering in Medicine and Biology Society Conference (EMBC), 2014
[pdf] [bibtex] [slides]
- Ajay Kumar Tanwani, Aude Billard. Transfer in Inverse Reinforcement Learning for Multiple Strategies. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013
[pdf] [bibtex] [video] [slides]
- Ajay Kumar Tanwani, Aude Billard. Inverse Reinforcement Learning for Compliant Manipulation in Letter Handwriting. National Center of Competence in Robotics Research (NCCR) Review, 2013
[poster]
- Pawel Wawrzynski, Ajay Kumar Tanwani. Autonomous reinforcement learning with experience replay. Neural Networks, Elsevier, 2013
[pdf] [bibtex]
- Ajay Kumar Tanwani. Optimizing walking of a humanoid robot using reinforcement learning. MS Thesis, European Masters on Advanced Robotics (EMARO), Nov 2011
[pdf] [bibtex] [video]
- Ajay Kumar Tanwani, Muddassar Farooq, Classification potential vs classification accuracy: a comprehensive study of evolutionary algorithms with biomedical datasets. J. Bacardit et al. (Eds), Learning Classifier Systems (LCS), Springer LNAI, 2010
[pdf] [bibtex]
- Ajay Kumar Tanwani, Jamal Afridi, Zubair Shafique and Muddassar Farooq. Guidelines to select machine learning scheme for classification of biomedical datasets. 7th European Conference on Evolutionary Computation, Machine Learning and Datamining (EvoBIO), Springer LNCS, 2009
[pdf] [bibtex]
- Ajay Kumar Tanwani, Muddassar Farooq. The role of biomedical dataset in classification. Artificial Intelligence in Medicine (AIME), Springer LNAI, 2009
[pdf] [bibtex]
- Syed Bilal Mehdi, Ajay Kumar Tanwani, Muddassar Farooq. IMAD: In-execution malware analysis and detection. Genetic and Evolutionary Computation Conference (GECCO), 2009
[pdf] [bibtex]
- Ajay Kumar Tanwani, Muddassar Farooq. Performance evaluation of evolutionary algorithms in classification of biomedical datasets. International Workshop on Learning Classifier Systems, IWLCS (GECCO), 2009
[pdf] [bibtex]
- Amina Tariq, Ajay Kumar Tanwani and Muddassar Farooq. User-centered design for mobile healthcare application in conservative environments. 10th Annual Conference of the NZ group on Human Computer Interaction, CHI-NZ, 2009
[pdf] [bibtex]