Deep Learning

HannibalRex

@topspin mine too. I know just enough to know when someone in my class has had too much DCNN Kool aid

BernieTheBernie

I mentioned the Python WTF some where above, didn't I?
Well, just downloaded and installed the latest Python 3.7.4 and tested it (open a Windows command line window for that, not the Python console!):

Thank you.
(Oh by the way, the place where I found the commands is https://www.tensorflow.org/install/pip? )

BernieTheBernie

Next step:

It is an Administrative command line, and still fails.
Python is one of the Greatest WTFs available!

Adynathos

It is easier to install and use python with conda:

Miniconda — Anaconda documentation

Conda also has its own package repository with more native libraries (where PIP is usually limited to the python libs).

dkf

@Adynathos It's not usually a big deal on platforms that have their Big Boy Pants on, as they can build native libraries from source as required.

Adynathos

I also recommend Jupyter as a nice GUI for running python programs with images

Installation — JupyterLab 4.2.1 documentation

As for neural nets, there are 2 feasible frameworks as of now: PyTorch and Tensorflow.
I use PyTorch and it is quite straightforward.
Tensorflow's current version 1.* has several different APIs for the same thing so it might be confusing. There is a TF 2.0, I heard they removed some APIs and left one, but not sure if its a sane one.

Caffe, which i saw in that book, is bad - I tried running code published for some articles, and when the code was on Caffe, it always failed to compile.

dkf

@Adynathos said in Deep Learning:

I also recommend Jupyter as a nice GUI for running python programs with images

QF

BernieTheBernie

Next step. After a reboot of the computer, I could install virtualenv. Now comes tensorflow. Their webpage says:

Verify the install:
(venv) $ python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

So I did:

Gaahhh!!! Long lines of messages. Errors?
No, only warnings.
Did it work eventually?

Yes, it did:

Tensor("Sum:0", shape=(), dtype=float32)

Great.

BernieTheBernie

With the Python installation, a shell called "IDLE" was included. I just tried it.
At least, copy and paste works far better than in the "common" shell. But the execution of a training takes eternaties, due to some special characters being printed which IDLE does not interpret usefully:

Python is Great!

_P_

@cvi Deep learning basically a glorified form of pre-computation, except that you throw tons, tons and tons of computational power at this part because we have too much computational power (just look at crypto mining). Then actually running the model spends much less time.

And I think the deep learning frameworks and the language used by them (aka network connection, dropout, etc...) form the class of 5GLs. It's like when SQL claimed "you don't have to specify how to do something, just specify what to do", except it's actually (kinda) true, and you don't need an engine tuned by many talents for decades anymore. And that's the hype around deep learning: if you throw enough computational power at it, design an acceptable model that is big enough, and train it properly, it will probably solve any kind of problems better than imperative algorithms.

And there are many problems with it, which leads to various jokes and implications:

Deep learning is very lazy: it only learns the minimal amount of features needed and uses the quickest, dirtiest way to complete the task. In a sense it's the greatest meta troll ever. And it has trolled us in many ways already
Deep learning can solve almost every problem, but it's very costly. Some people seem to be actually serious when they want to try using deep learning to solve fizzbuzz or hello world, which is... as funny as mining crypto, I guess?
As a continuation of point 1, most uses of deep learning that are actually successful are on very constrained toy problems, because they have well-defined rules and does not have loopholes easily abused by deep learning algorithms. Yup, just like a junior coder writing simple code that doesn't handle all the complex real-world/business logic cases. It's waiting to crumble to dust as soon as it touches the unfortunate thing called real life.
More generally, we don't actually know how to design an AI for the most general application while also preventing it from doing stupid, ridiculous things that can easily cause harm. It's a hot field called AI safety. Robert Miles had lots of videos on youtube discussing this can of worms.

topspin

@BernieTheBernie said in Deep Learning:

With the Python installation, a shell called "IDLE" was included.

Kill it with fire!

Seriously, get something IPython based (IPython itself, Jupyter Qt console, Jupyter notebook, ...) for a shell, or basically anything else for an IDE. (Spyder, PyCharm, Atom with hydrogen, VS Code I think, ...). Last time I touched IDLE it was just incredibly bad.

Python is actually quite nice when you got all the bullshit out of the way. I never got around to understanding all the pip and virtualenv and conda and ... crap. If you got the space to spare just install the Anaconda distribution with everything and stuff mostly just works out of the box.

Captain

@HannibalRex said in Deep Learning:

@pie_flavor I don't remember the exact context, but another student in a discussion class of mine said, in all seriousness, that deep neural networks should be trained by other deep neural networks. We ended class.

Nothing wrong with adversarial neural networks...

_P_

@Captain said in Deep Learning:

@HannibalRex said in Deep Learning:

@pie_flavor I don't remember the exact context, but another student in a discussion class of mine said, in all seriousness, that deep neural networks should be trained by other deep neural networks. We ended class.

Nothing wrong with adversarial neural networks...

Can we train our children with adversarial neural networks to replace the mouth-breathing parents?

BernieTheBernie

@topspin said in Deep Learning:

Last time I touched IDLE it was just incredibly bad.

When was that? Just had to find out that there is an old open bug:

Issue 1442493: IDLE shell window gets very slow when displaying long lines - Python tracker

Worth noting: "Created on 2006-03-03 14:45".
Yes, already more than 13 years old.

BernieTheBernie

@Adynathos said in Deep Learning:

recommend Jupyter

@Adynathos said in Deep Learning:

easier to install and use python with conda

@Adynathos said in Deep Learning:

I use PyTorch and it is quite straightforward

@topspin said in Deep Learning:

get something IPython based

Thanks for the hints. But note that I am actually not interested in the Python part, but in the Deep Learning part. I'd really prefer it if I could just use C#. The more extras I have to use, the more things will look different from the examples given in text books, and thus add another layer of confusion.
Confusion is big anyway, because of tyops or other errors in the books.

Yeah, are those networks really "convolutional" networks or rather "confusional" networks? I'll try to find out. Seriously.

dkf

@levicki said in Deep Learning:

you really don't even need virtualenv

It depends what you're doing with it. A single application? Yes, you don't need virtualenv (or one of the other tools like it). More than one non-trivial application? You do need virtualenv as the applications are quite probably incompatible with each other, and Python doesn't really distinguish between versions of a library except when doing package installation. Those profiles for different applications are the real problem that virtualenv solves.

It's the Classic Python Way. Looks nice on the surface and a complete ghastly mess underneath.

dkf

@BernieTheBernie said in Deep Learning:

When was that? Just had to find out that there is an old open bug:

I recognise the underlying bug that causes that. Fixing it is… trickier than it sounds.

dkf

@BernieTheBernie said in Deep Learning:

I'd really prefer it if I could just use C#.

The problem is that the core people working on neural networks have, to date, really preferred working in Python, with that being the language that gets all features first and others are all using someone's hobby re-embedding of the (fucking huge!) API in another language. I looked once and rapidly thought of

topspin

@Captain

error

@dkf said in Deep Learning:

Python doesn't really distinguish between versions of a library except when doing package installation.

Well, it's still an emerging language and they're ironing out the kinks, I'm sure this showstopper bug will be fixed before they launch the big version one oh...

Wait, what's that you say...? It's how old?

topspin

@BernieTheBernie said in Deep Learning:

@topspin said in Deep Learning:

Last time I touched IDLE it was just incredibly bad.

When was that? Just had to find out that there is an old open bug:

Issue 1442493: IDLE shell window gets very slow when displaying long lines - Python tracker

Worth noting: "Created on 2006-03-03 14:45".
Yes, already more than 13 years old.

Not sure, but I guess around 2013. I doubt it’s improved since, though.

@dkf said in Deep Learning:

@BernieTheBernie said in Deep Learning:

When was that? Just had to find out that there is an old open bug:

I recognise the underlying bug that causes that. Fixing it is… trickier than it sounds.

Oh yeah, sorry to step on someone’s toe, but the whole thing being Tk based isn’t really speaking for IDLE, either.

_P_

@BernieTheBernie said in Deep Learning:

But note that I am actually not interested in the Python part, but in the Deep Learning part. I'd really prefer it if I could just use C#

That doesn't make much sense either because deep learning uses its own language to describe the type of each layer, connections, dropout... it's basically a different kind of "programming" language. Using tensorflow/pytorch/darknet... in Python is nothing but utilizing a wrapper, so even if you use C# it'll still be another wrapper, and you still have to lay out the entire network. And even though an argument could be had about whether coupling everything deep learning-related to Python is a good idea, it's much more convenient to work with with libraries like numpy at your disposal. Just imagine doing all of those data structure magic in other languages, in a clean and concise way.

cvi

@_P_ said in Deep Learning:

data structure magic

error

@cvi said in Deep Learning:

@_P_ said in Deep Learning:

data structure magic

He means arcane, unreadable syntax like array comprehensions.

dkf

@error said in Deep Learning:

He means arcane, unreadable syntax like array comprehensions.

The really incomprehensible stuff is when you do non-trivial data rearrangement using numpy.

_P_

@error said in Deep Learning:

@cvi said in Deep Learning:

@_P_ said in Deep Learning:

data structure magic

He means arcane, unreadable syntax like array comprehensions.

It's a thing called metaprogramming. Have you tried that?

error

I've written code that scaffolds code that generates code that builds code that tests my code after it transpiles my code.

Does it get more meta than that?

Gąska

@error

_P_

@error That sounds like something that'd help nobody else but yourself, in contrast to numpy.

topspin

@error said in Deep Learning:

@cvi said in Deep Learning:

@_P_ said in Deep Learning:

data structure magic

He means arcane, unreadable syntax like array comprehensions.

What? Array comprehensions are much more readable than map-filter-fold-whatever. Or LINQ, maybe?

_P_

@topspin said in Deep Learning:

@error said in Deep Learning:

@cvi said in Deep Learning:

@_P_ said in Deep Learning:

data structure magic

He means arcane, unreadable syntax like array comprehensions.

What? Array comprehensions are much more readable than map-filter-fold-whatever. Or LINQ, maybe?

Except that it has nothing to do with numpy, numpy's concern is easy, powerful manipulation of multi-dimensional data structures that also provides good performance. Now try to do that in other languages.

dkf

@_P_ said in Deep Learning:

numpy's concern is easy, powerful manipulation of multi-dimensional data structures that also provides good performance

Well, they don't really achieve “easy” (unless you're doing something that they've provided directly) but the rest is largely true. You probably can beat it by writing the manipulations in a fully compiled language and taking care over it but it's a reasonable approach for prototyping the data transforms.

topspin

@dkf said in Deep Learning:

Well, they don't really achieve “easy” (unless you're doing something that they've provided directly)

I find the "normal" operations you'd usually want (arithmetic, slicing, dot products, etc.) go far enough, even though I sometimes need to look up what to do. If you need a reduction that's complicated enough you end up using einsum, well, you're pretty much screwed.

dkf

@topspin said in Deep Learning:

I find the "normal" operations you'd usually want (arithmetic, slicing, dot products, etc.) go far enough, even though I sometimes need to look up what to do. If you need a reduction that's complicated enough you end up using einsum, well, you're pretty much screwed.

This is from our code; try not to vomit on seeing it:

times, indices = numpy.where(numpy.fliplr(numpy.unpackbits(
    raw_data[:, 1:].byteswap().view("uint8")).reshape((-1, 32))).reshape((-1, n_bytes * 8)) == 1)

It's not the whole of the transform either; raw_data is still a numpy array (of int32), but the code to generate it isn't quite so horrid.

topspin

@dkf said in Deep Learning:

try not to vomit on seeing it

EDIT: But you'll probably not express that "simply" in another language, either.

topspin

@levicki It flips, unpacks, byte-swaps, reshapes, wheres, and reshapes. I mean, Duh.

error

@topspin said in Deep Learning:

@levicki It flips, unpacks, byte-swaps, reshapes, wheres, and reshapes. I mean, Duh.

Topspin is channeling ghostdoc.

Filed under: The product that generates useless comments that just repeat what the code already says.

error

@topspin said in Deep Learning:

But you'll probably not express that "simply" in another language, either.

I'd argue that if your operation is intrinsically complex, then it should verbosely express each individual step in a way that accurately reflects that complexity, rather than try to fit an entire algorithm onto one line (which is next to impossible to debug or to decipher). Not that you can debug or decipher machine learning anyway.

dkf

@levicki said in Deep Learning:

What the fuck does that code do?!?

It unpacks a matrix of bits that describe when events happen and converts them into a much larger matrix of timestamps of those events that occurred. The bit-per-event representation is extremely compact at the typical event rates we're dealing with, and much faster to write on the other side where that's one of the many bottlenecks in the simulator.

Our attitude to that code is: open it up, take a look around, verifies that it does what it is supposed to, then close it back up again and pray to not need to look again.

_P_

@dkf And that's the point: if there is a tight loop about this kind of low level data transformation, you most likely don't need to touch it in the first place so there is not really much need for "maintenance" or whatever.

And yet there are always people yelling at these kind of code as Unreadable, Unmaintainable, Not for Production (TM) regardless of actual code, and I just love them. After all, if you manually unroll everything in imperative loops, it must be more readable and maintainable, right? And to think that most actually unreadable code frequently featured in TDWTF are actually those imperative code Not to mention that this code is actually pretty readable (talking about less than a minute there) if someone puts a line of comment describe its effect. Unless someone is taking the code out of context trying to prove something, but then that person would be .

In any case, 3/10 the bait is salty

dkf

@_P_ said in Deep Learning:

if there is a tight loop about this kind of low level data transformation

It describes a tight loop. Its input data is a matrix built out of the raw data, which has a somewhat circuitous path to reach this point. (We've a similar thing for unpacking the history traces of the state variables of the simulation, but that's a bit less horrible despite needing to convert fixed point to float; it's unpacking all those bit flags that is nasty.) We're planning to change it so that we end up with keeping the unpacked information (plus the metadata describing what was actually simulated) in a database; the actual raw data is already in the DB already (because then we don't have the headache of managing tens of thousands of working files) but there's some critical composition steps required to make everything come together.

The matrices are often hundreds of megabytes of raw binary data. Their manipulation is why we build servers for running Python with 1TB or more of RAM…

BernieTheBernie

Next step. Having installed Python 3.7.4 and Tensorflow, I try t run some examples in the book. Works wonderful, of course: there is no "mul" method in Tensoflow, use multiply. There is no substract method, it is called subtract. There is not train.SummaryWriter, use summary.FileWriter instead. Etc.

Eventually, a graph is exported and now you should be able to view it in Tensorboard.
So I type in jupyter notebook:

tensorboard --logdir="C:\\Freigabe\\Temp\\my_graph"

And the error message is:

File "<ipython-input-46-b4e562fe914f>", line 1
   tensorboard --logdir="C:\\Freigabe\\Temp\\my_graph"
                                                   ^
SyntaxError: can't assign to operator

OK, then I learned that I have to execute it on the (Windows) command line, or add an ! in jupyter. And there is a command not found error. Great.

So I browse the internetz to find a solution. OK, I could install Tensorboard. Install bazel in order to do so. Install msys2 in order to install bazel. WTF? Well, worse than that:

That crappy crap is treating my machine as Windows 95 device.
After that I could start bazel.

And? The error is: not enough system resources.

OK, there are more ways to start Tensorboard. At least, I found that there is a folder with that name (though no executable...).
cd \ c:\>python c:\Users\BernieTheBernie\AppData\Roaming\Python\Python37\site-packages\tensorboard\main.py --logdir="C:/Freigabe/Temp/my_graph"

When opening http://localhost:6006/ in Internet Explorer or Firefox, I see an empty page only (but it's tons of javascript, I looked at the source).

Only in Chrome, I see the expected graph.

Oh great. Deep Learning is so professional....

I won't call that even beta.

dkf

@BernieTheBernie Sounds like the package scripts weren't in the path, or perhaps you weren't running the virtualenv right (since it looks like an application in a Python package; that's a thing). Debugging these things can be a bit mysterious, especially when the original instructions were developed on a different platform and the whole thing ends up falling into the cracks in Python's platform-independence.

Deep Learning is so professional....

So professional. Much cross platform. Wow.

BernieTheBernie

Next little script:

import matplotlib.pyplot as plt
from PIL import Image

ImportError: No module named matplotlib.pyplot
OK, that was easy: pip install matplotlib

Next:
ImportError: No module named PIL
Taking a look at https://stackoverflow.com/questions/8863917/importerror-no-module-named-pil , i tried easy_install PIL. That failed, of course.

Searching for PIL
Reading https://pypi.org/simple/PIL/
No local packages or working download links found for PIL
error: Could not find suitable distribution for Requirement.parse('PIL')

Hmpf.
pip install image worked, and then I had to use import image (lower-case i!) instead of from PIL import Image

_P_

Honestly at this point it is more about Python library usage more than actual Deep Learning (TM). I expected more from the original WTF.

remi

@BernieTheBernie said in Deep Learning:

There is no substract method, it is called subtract

Brillant!

cabrito

@BernieTheBernie said in Deep Learning:

Next little script:
import matplotlib.pyplot as plt
from PIL import Image
ImportError: No module named matplotlib.pyplot
OK, that was easy: pip install matplotlib

Next:
ImportError: No module named PIL
Taking a look at https://stackoverflow.com/questions/8863917/importerror-no-module-named-pil , i tried easy_install PIL. That failed, of course.
Searching for PIL
Reading https://pypi.org/simple/PIL/
No local packages or working download links found for PIL
error: Could not find suitable distribution for Requirement.parse('PIL')
Hmpf.
pip install image worked, and then I had to use import image (lower-case i!) instead of from PIL import Image

PIL (Python Image Library) was superseded by pillow (which at least first years was a drop-in replacement, no need to touch imports)
Your pip install image grabs a Django util to crop resize, etc images, which probably uses and install pillow under the hood, "fixing" your import problem

dfdub

@BernieTheBernie said in Deep Learning:

Next step. After a reboot of the computer, I could install virtualenv.

In Python 3, you actually don't really need virtualenv, since venv is available OOTB.

topspin

@dfdub said in Deep Learning:

@BernieTheBernie said in Deep Learning:

Next step. After a reboot of the computer, I could install virtualenv.

In Python 3, you actually don't really need virtualenv, since venv is available OOTB.

We've also witnessed here installation with pip, easy_install, conda was mentioned...
I thought TMTOWTDI was frowned upon as the Perl way?!

dkf

@topspin said in Deep Learning:

I thought TMTOWTDI was frowned upon as the Perl way?!

It's frowned on, but Python's full of lots of ways to do things anyway. Sometimes, programming in it is about figuring out how to trick the libraries into working together…