Sebastin Santy

I am a research intern at Carnegie Mellon University, where I work on computer vision under the supervision of Prof. David Touretzky.

Previously, I was a research intern at University College London working on Information Retrieval under the supervision of Prof. Emine Yilmaz and Dr. Rishabh Mehrotra. I have also been a Google Summer of Code (GSoC) developer for 2 consecutive years: 2018 with NumFOCUS (Julia Computing) and 2017 with Mozilla (Bugzilla). I am a senior undergraduate from BITS Pilani, Goa, India.

Some of the work I did at UCL is documented at UserContext.

I am a tetraglot - I can speak English, Hindi, Malayalam and Gujarati fluently.

Email  /  CV  /  Google Scholar  /  Github  /  LinkedIn  /  Twitter


I'm interested in computer vision and information retrieval.

Towards Task Understanding in Visual Settings
Sebastin Santy, Wazeer Zulfikar, Rishabh Mehrotra, Emine Yilmaz
AAAI Conference on Artificial Intelligence, 2019
project page / paper

Understanding tasks being undertaken in a scene. Leveraging insights from real world task understanding systems, to propose a framework composed of convolutional neural networks, and an external hierarchical task ontology to produce task descriptions from input images. Check UserContext for more.

Leveraging Task Information for Question Recommendation
Sebastin Santy, Rishabh Mehrotra, Emine Yilmaz
Under Review

Leverage an existing hierarchical task ontology in order to understand user behaviour from the task perspective. A lot of daily searches performed involve a user being interested in accomplishing a particular task. Hence, leveraging task information will help in question recommendation.

DataDepsGenerators.jl: making reusing data easy by automatically generating DataDeps.jl registration code
Lyndon White, Sebastin Santy
Journal of Open Source Software, 2018
paper / blog post / code

DataDepsGenerators.jl is a metadata generating Julia package for data dependency management. It leverages many different APIs to support a very large number of repositories. By current estimates tens of millions of datasets are supported, from hundreds of repositories.

BITS Darshini: A Modular, Concurrent Protocol Analyzer Workbench
Prasad Talasila, Mihir Kakrambe, Sebastin Santy, Anurag Rai, Neena Goveas, B. Deshpande
ACM International Conference on Distributed Computing and Networking, 2018
paper / code / website

Network measurements are essential for troubleshooting and active management of networks. Protocol analysis of captured network packet traffic is an important passive network measurement technique used by researchers and network operations engineers. In this work, we present a measurement workbench tool named BITS Darshini (Darshini in short) to enable scientific network measurements.


In 2010, I had the opportunity to meet and talk with His Excellency Dr. APJ Abdul Kalam, Former President of India, popularly known as the Missile Man of India. We talked on extensive issues regarding science and education in India. This talk was possible as a result of me winning a national competition - Iken Scientifica. The entire show was aired on National Geographic Channel. Check here.

I consider myself privileged to have had access to my dad's electronics/mechanical workshop, where I used to fiddle with stuff lying around. In 2012, during my 9th grade, I hacked up an autonomous robot to be used on farms. This autonomous robot was able to plough, sow and cover the soil without any human labour. It was aimed towards diminishing the latent labour prevalent on Indian farms.

(Jon made this nice website template)