On RAI, access neutrality, and how to download Smooth Streaming content

by ao2, December 27, 2010

RAI, the Italian public radio/TV broadcaster, is using Silverlight and Smooth Streaming to serve most of the media content on its website, Smooth Streaming is the Microsoft interpretation of Adaptive Streaming technology over HTTP, which was firstly introduced by Apple for its mobile platform; check out the IIS Smooth Streaming Technical Overview for further details on Smooth Streaming.

Adaptive streaming

Adaptive streaming technology is a quite smart idea, the client adapts the content quality to its resources (CPU, network bandwidth or latency) so to achieve a smoother viewing experience (it does so in its own interest when real-time fruition is the goal), the content server benefits from that too as it achieves a sort of “distributed load balancing” for free, distributing loss of quality among all the clients as the service degrades; unfortunately it looks to me that open implementations of these mechanisms are still not on par with closed source ones, this is true at least for Smooth Streaming. There are also some recent discussions about adaptive HTTP streaming for open codecs, but this is a problem orthogonal to open implementations of already existing mechanisms.

Rai.it and Smooth Streaming

I am not going to criticize the choice of RAI to use Silverlight as a Media Platform in this post because I don't know the platform very well and how much closed it is, but I know that the Open Source implementation of it called Moonlight still lags behind the Microsoft one, in particular Moonlight does not support Smooth Streaming yet, although there are plans to support it as we can see in the Roadmap.

The bottom line is: on systems where Silverlight is not (fully) working, most of the new content on the RAI website is basically unaccessible, and this is not acceptable (unaccessible is unacceptable someone would say): as a Public Service broadcaster, they should provide a service as neutrally accessible as possible.

An ad interim solution is provided by a hacky script I put together: you can find it in the smooth-dl git repository it allows content served via Smooth Streaming to be downloaded and viewed off-line; consider this an early and faulty implementation (it only works for RAI Manifests so far) used to understand how the Smooth Streaming mechanism works, modulo the adaptive part; so patches are more than welcome.

smooth-dl has been inspired by ismdownloader, whose author also gave me some tips —thanks—, but ismdownloader only works in Windows and is closed source, so it was not an option for me.

In order to use smooth-dl, for now, you have to retrieve the client manifest URL by scraping through the web page (looking for the mediaUri value) after having spoofed a Silverlight installation with a Greasemonkey userscript like the following (this makes the player <object></object> element to be rendered):

unsafeWindow.Silverlight.isInstalled = function(version) {
  return true;
};

Maybe this can be integrated in other userscripts like Rai.tv native video player for programmes and RAI-TV-Player (or better, implement the equivalent effect using location hack). I also think all the RAI-accessibility related efforts should be merged together into a single project, the olrait mailing list looks like a good starting point.

Is it too much asking for the direct link to the content (or its manifest) to show up when the intended player is not available on the User Agent? This way users would be free to access the programs the way they want. Really, please, RAI, be more open and expose those links yourself, you're a public service after all, don't let us out of your site (pun intended).

Long term solution

Only after I wrote smooth-dl I found libsmth which looks more promising as a base for a long term solution to the problem of an open Smooth Streaming client, but it needs more people working on it, so please spread the word, if you are interested in having Smooth Streaming working in your favourite Open Source media player let's collaborate on libsmth, its author Stefano Sanfilippo is willing to take contributions. Check out the libsmth Ohloh page as well for some statistics about it.

Short link: https://ao2.it/50

Similar By Terms

Comments

by Anonymous (not verified) | Fri, 13/05/2011 - 04:10

Mi interessa moltissimo

Mi interessa moltissimo quello che stai facendo.
Anche se è una soluzione provvisoria, come dici, mi piacerebbe comunque tentare di scaricare programmi Rai ( magari quelli di Rai- replay) con il tuo smooth-dl. Potresti spiegare come si fa ad una UTONTA? Con parole semplici semplici?
Grazie

by ao2 | Mon, 16/05/2011 - 00:40

Su sistemi Windows forse, per

Su sistemi Windows forse, per ora, è più semplice usare ismdownloader, trovi le istruzioni in questo commento sul blog di Flavio Tordini e nei commenti successivi.

Su sistemi GNU/Linux smooth-dl è facile da usare se hai già installato python, scarica smooth-dl da qui: smooth-dl.py.

E segui le istruzioni nel file README.

Il punto è che non so se valga la pena impiegare tempo a scrivere una guida dettagliata in Italiano per utenti finali dato che il programma non è ancora completo, l'ideale sarebbe avere un utente più esperto che scriva i passaggi fatti al suo primo utilizzo.

by piero (not verified) | Fri, 01/07/2011 - 12:27

Hi, Moonlight implements

Hi, Moonlight implements Smooth Streaming today with Monlight 3 preview and this version works with Replaytv content. Rai not discriminate any platform. Silverlight and Smooth Streaming is one of the best technology for streaming content over Internet like also you write in your article. Why this is not good for you ? This is not rue that Apple Streaming arrived befor Smooth Streaming. Smooth STreaming come to the market before Aplle streaming and is the first credible adaptive streaming technology for massive use. Specification of Smooth Streming protocolo and file format was released (http://learn.iis.net/page.aspx/684/smooth-streaming-transport-protocol). The first adaptive streaming technology come from Move Network but it is too complex and required a thousand of framented files fro every asset (like in Apple implementation) and it is very diffucult to use for a large use. Your post is not complaiant with the current law about copyright and I suggest to you to remove it. Rai is a public broadcaster but not all the content that Rai broadcast is owned by Rai. Rai broadcast content like movies, fiction, shows that is not produced by Rai but from others company and Rai have only the rights for broadcast on Rai channel and streaming on Internet "ONLY IN STREAMING FROM RAITV PORTAL" your post and your script violate this and is not complaiant with the current law and you can have legal problem if you promote illegal download .

by ao2 | Fri, 01/07/2011 - 22:38

@piero: I'll try to fix the

@piero: I'll try to fix the inaccuracies about what Adaptive Streaming technology came first, thanks for pointing them out.

About the legal matters I don't have the knowledge to comment on that TBH, I understand that Rai itself (the content server part I mean) may only have the permission to stream some content and not to serve it for download, but from a technical point of view a client just gets fragments from the stream server, now whether it renders them or stores them is none of the server concern.

But law is not always logical, we all know that; stealing a random quote from Lawrence Lessig in one of his talks:

Common sense, a rare idea in the law

Just for the record: I am not promoting anything, I just tried to work around a problem I experienced myself.

Regards, Antonio

by mitm (not verified) | Fri, 01/07/2011 - 17:46

articolo interessantissimo,

articolo interessantissimo, grazie per aver condiviso il tuo lavoro!

su Olrait, la ML che citavi, abbiamo pensato di scrivere un'estensione per firefox che consente la visualizzazione dei contenuti Smooth Streaming (live/vod e replay) e MMS dal portale rai.tv, attraverso mplayer, player nativo su molte distribuzioni linux.
Il componente aggiuntivo, che abbiamo ribattezzato raismth, è scaricabile da addons.mozilla.org

@piero: moonlight does-not-work! hundreds of users complain about it and nowadays it's not compatible with firefox 5...

by ao2 | Fri, 01/07/2011 - 22:42

@mitm: c'è un posto da cui

@mitm: c'è un posto da cui scaricare il codice di Raismth? Magari potreste metterlo sulla pagina launchpad di olrait, o magari su gitorious.org.

Non ho mai visto come si scrive una estensione per Firefox e compagni e questa potrebbe essere una buona occasione.

Grazie, Antonio

by mitm (not verified) | Sat, 02/07/2011 - 10:52

@ao2: ciao antonio, i

@ao2: ciao antonio,
i componenti aggiuntivi per firefox sono pacchettizzati e distribuiti come zip file, come puoi vedere dalle specifiche
Per cui non e' necessario 'pubblicare' il sorgente, in quanto spacchettando il file xpi già lo hai a disposizione.

Tra l'altro quando installi un'estensione, se la preferenza di mozilla
extensions.alwaysUnpack
è settata a true (vedi about:config), nel profilo utente di firefox l'archivio xpi viene già scompattato, per cui i "sorgenti" sono a portata di mano.

RAISMTH è scritto in javascript/xul molto "rudimentale", come avrai modo di vedere..
essendo questo il nostro primo approccio allo sviluppo di componenti per firefox, il codice è molto sporco, tuttavia è funzionale al suo scopo

L'estensione è stata pubblicata su addons.mozilla.org ed è in attesa di una revisione preliminare
il codice è direttamente consultabile via web
La licenza GPL v.3

by ao2 | Sat, 02/07/2011 - 11:47

@mitm, visto, grazie; quindi

@mitm, visto, grazie; quindi tutto il lavoro viene fatto in overlay.js, ed è ovviamente simile a ciò che fa smooth-dl; e vedo anche che entrambi i progetti mancano di generalità nel parsing del Manifest :)

Comunque mettere il codice sotto controllo di revisione è utile anche per tenere traccia della storia dello sviluppo del progetto e per consentire a più persone di lavorare sul codice in modo più comodo.

Ciao, Antonio

by mitm (not verified) | Sat, 02/07/2011 - 12:50

esatto Antonio, overlay.js è

esatto Antonio, overlay.js è il fulcro dell'addon, il codice ricalca uno script bash di kowalsky di cui già esistono port in php e perl

in merito al parser dei manifest, contiamo di inserirne uno "decente" non appena risolti alcuni problemi con i .csm di rai replay (che ci risultano non well-formed)

per quanto riguarda il discorso del controllo di revisione del codice, sono perfettamente d'accordo, ma visto il livello di partecipazione al progetto su olrait, non credo ci siano i presupposti...

voglio precisare che quello che ci proponiamo, per il momento, non è creare un supporto completo allo smooth streaming su linux, quanto fornire una soluzione accettabile alle evidenti lacune di moonlight e al totale menfreghismo di Rainet, che penalizzano, ancora una volta, chi sceglie sistemi e software liberi..

ciao, Riccardo

by evilripper (not verified) | Thu, 14/07/2011 - 11:51

grazie per le info! sei un

grazie per le info! sei un mito! :-)
è uno schifo che la rai usi silverlight ed è uno schifo che non permetta ai cittadini di usarli come vuole! :-(

by Anonymous (not verified) | Sat, 19/09/2015 - 01:09

I'm trying to use your

I'm trying to use your script, but there is an error like:

./smooth-dl.py -v 0 -a 0 -q 0 -Q 0 -d ./ "http://ism_url_is_here"

Stream: 0 Type: video
	Quality Levels:
	 0: H264   1280x720 @ 1429486 bps
	 1: H264    960x540 @  856328 bps
	 2: H264    480x270 @  462836 bps
Stream: 1 Type: audio
	Quality Levels:
	 0: AACL 44100Hz 16bits 2ch @   95998 bps
	 1: AACL 44100Hz 16bits 2ch @   95998 bps
	 2: AACL 44100Hz 16bits 2ch @   95998 bps


Downloading Stream 0
	Downloaded size: 27622341    

Downloading Stream 0
	Downloaded size: 27622341    

Rebuilding Stream 0
Traceback (most recent call last):   
  File "./smooth-dl.py", line 430, in 
    options.chunks_dir, options.download)
  File "./smooth-dl.py", line 342, in smooth_download
    chunks_dir, dest_video)
  File "./smooth-dl.py", line 261, in rebuild_stream
    data = get_chunk_data(f.read())
  File "./smooth-dl.py", line 50, in get_chunk_data
    mdat_size = struct.unpack(">L", data[moof_size:moof_size + 4])[0]
struct.error: unpack requires a string argument of length 4

Do you have any idea?

by ao2 | Sat, 19/09/2015 - 12:24

I think smooth-dl does not

I think smooth-dl does not support the data format of the stream you are trying to download.

Maybe if you give me (even privately) the URL of the problematic files I could try to fix that.

by Anonymous (not verified) | Wed, 13/01/2021 - 14:28

hi hello. it seems like the

hi hello.
it seems like the project needs a lil update (:
just wondering will there be any updates?
the issue i am facing is that the files are not merging very well and are unplayable

i am interested in downloading this video https://playready.directtaps.net/smoothstreaming/SSWSS720H264/SuperSpeed...
url: https://developers.canal-plus.com/rx-player/
and open-sourced at https://github.com/canalplus/rx-player
Thank you for your time

by ao2 | Tue, 13/04/2021 - 22:38

Hi, I think nowadays we

Hi,

I think nowadays we would have better luck to cook somethig based on GStreamer, for example I can play the URL using gst-play-1.0

ao2.it
Antonio Ospite, theorist attacks

Interface language