Starting a Startup

It’s been since last Feb, that I’ve been working with two friends on what we would like to become a full-fledged startup. Since we are all busy with our own lives, we have been working on this project honestly too little for my taste. Since as well I have to control my tendencies to follow blindly my enthusiasm, I’ve been isolating the hours I allow myself to work on it on Sats.

Read More

Bot with Big Personality

I’m working on a bot that has to keep a consistent personality while talking about anything. What I have now is a bot that can be given a description of an environment, a description of which personality it should take, and the sentence that a human is asking to it. For now it is able to provide varied replies. The language it uses is not yet the sharpest possible but I think it’s fun to interact with.

Read More

Take a Derivative wrt the Moon

For the article I’m working on, I needed to control some properties of the derivative of the hidden state of a recurrent network wrt to the hidden state at the previous time step, basically

Read More

Our Workshop at NeurIPS

What started as wishful thinking between like minded friends, ended up as a complete Workshop in what is probably the most important scenario for Artificial Intelligence, NeurIPS. They realized not enough attention was being paid to the emerging field that lies in between deep learning (DL) and differential equations (DE) and they wanted to provide a platform for that scientific community to be able to discuss and share ideas. In recent years, there has been a rapid increase of machine learning applications in computational sciences, with some of the most impressive results at the interface of those two fields.

Read More

TensorBoard for Grads and Individual Weights

I needed to record gradients during training and individual weights, instead of tensor means, using tensorboard, but the former is not available anymore by default, and the latter never was. So I updated the tf2 callback, and since probably some of you might find it useful you can find it here.

Read More

A chatbot to talk about the real stuff [Ep. 2]

I wanted to complete the previous post where I reproduced in tensorflow one of the generative chatbots proposed by Wizard of Wikipedia: Knowledge-Powered Conversational agents. Funny enough, they linked my project in their official repository (check the Note: an unofficial …), which could not make me more proud! To be honest I asked if they could, but they did! As well I mentioned in the previous post that I didn’t know if they optimized on the perplexity (PPL) or masked perplexity, and they confirmed by email that they optimized on PPL, which is good news for me, since my results are better on PPL, not when I compare my masked PPL with their PPL.

Read More

ReQU might not be the activation you want

The book The Principles of Deep Learning Theory proposes in chapter 5 a method to study the quality of activation functions. The network they study is a fully connected network. They use the assumption that a desirable property for a network is that the covariance of their activations remains constant through the network, neither exploding nor imploding. They use this assumption/desiderata, to be able to use fixed point analysis to study how well a given activations encourages such representations.

Read More