&RPSUHVVLRQ��3URFHVVLQJ��,QGH[LQJ�DQG�5HWULHYDO�RI��'�2EMHFWV�DQG�'DWD�
+RZ�WR�H[WHQG�LPDJH�YLGHR�SURFHVVLQJ�WR�JUDSKLFV"
Tsuhan ChenCarnegie Mellon University
Joint work with Howard Leung, Masa Okuda, and Cha Zhang
7VXKDQ�&KHQ
�0LV�8QGHUVWDQGLQJ7R�JUDSKLFV�DQG�YLVLRQ�FRPPXQLWLHV9LGHR�LV�MXVW�ORZ�OHYHO SURFHVVLQJ
7R�WKH�YLGHR�FRPPXQLW\*UDSKLFV�LV�MXVW�VRPH�IDQF\�WRROV9LVLRQ�LV�WKLQJV�WKDW�GRQ¶W�ZRUN�LQ�SUDFWLFH
7VXKDQ�&KHQ
)LUVW�$WWHPSW«03(*��6WDUWHG�RXW�DV�PRGHO�EDVHG�FRGLQJ$QDO\VLV�DQG�V\QWKHVLV8VLQJ�YLVLRQ�JUDSKLFV�IRU�YLGHR�FRGLQJ
7KDW�GLGQ¶W�KDSSHQ��QRW�FRPSOHWHO\�6HWWOHG�ZLWK��'�VKDSH�EDVHG�FRGLQJ0RGHO�EDVHG�FRGLQJ�IRU�OLPLWHG�FRQWHQW��H�J���IDFHV
0RGHOLQJ�DQG�&RGLQJEXAMPLESMODELS CODED INFORMATION
PCM
Predictive CodingTransform Coding
Block-based codingH.261/263, MPEG-1/2
Model-based coding
MPEG-4
Pixels
Statistically dependent pixels
Moving blocks
Moving objects
Facial models
Moving regions
Color of pixels
Prediction error ortransform coeffs
Motion vectors and prediction error
Shapes, motion, and colors of objects
Action units
Shapes, motion, and colors of regions
Region-based codingH.263+, MPEG-4
MPEG-7A/V objects Description
7VXKDQ�&KHQ
0RGHOLQJ�DQG�&RGLQJ��FRQW��%HWWHU�PRGHOLQJ�LPSOLHV+LJKHU�FRPSUHVVLRQ0RUH�FRQWHQW�DFFHVVLELOLW\0RUH�FRPSOH[LW\/HVV�HUURU�UHVLOLHQFH
9LGHR�DQG�YLVLRQ�JUDSKLFV�GR�JR�KDQG�LQ�KDQG�DOO�DORQJ9LGHR�UHVHDUFK�LV�HYROXWLRQ�RI�YLVLRQ�DQG�JUDSKLFV�WHFKQLTXHV
7VXKDQ�&KHQ
7RSLFV&RPSUHVVLRQ�IRU�LPDJH�EDVHG�UHQGHULQJ
&RPSUHVVLRQ�IRU��'�PHVKHV6WUHDPLQJ�LQ�WH[WXUH�DQG�JHRPHWU\�MRLQWO\�
,QGH[LQJ�DQG�UHWULHYDO�RI��'�REMHFWV
%XLOGLQJ�LPPHUVLYH�HQYLURQPHQWV
7VXKDQ�&KHQ
,PDJH�%DVHG�5HQGHULQJ
««
>6KXP HW��DO@
7VXKDQ�&KHQ
&RPSUHVVLRQ
7KH�QXPEHU�RI�LPDJHV�LV�ODUJH��VR�ZH�QHHG�FRPSUHVVLRQ
*RRG�WR�KDYH�IHZHU�VDPSOHV'RHV�QRW�JXDUDQWHH�IHZHU�ELWV
&RQVLGHU�WKHVH�DV�D�YLGHR�VHTXHQFH*HQHUDO�YLGHR�FRGLQJ�DSSOLHV
7VXKDQ�&KHQ
DCT Q
IDCT
IQ
D
ME
MCMV
+
−�
�
IDCT
IQ
�D MC
MV
Network or Storage
Encoder Decoder
9LGHR�&RGHF
� ,QWUD�FRGLQJ� ,QWHU�FRGLQJ
�
� �
�
� �
Previous frame(reference frame)
Current frame
7VXKDQ�&KHQ
,QWUD�&RGLQJ
'LVDGYDQWDJH��'RHV�QRW�H[SORLW�WKH�FRUUHODWLRQ�EHWZHHQ�LPDJHV
,�IUDPH ,�IUDPH
««
� � �,�IUDPH
7VXKDQ�&KHQ
,QWHU�&RGLQJ
'LVDGYDQWDJH��'RHV�QRW�SURYLGH�UDQGRP�DFFHVV
L�H���IUDPH�1�GHSHQGV�RQ�IUDPH�1��
,�IUDPH 3�IUDPH
««
� � �3�IUDPH
7VXKDQ�&KHQ
3UHGLFWLRQ�IURP�6SULWH
��
««««
� � N�
>FI� $QDQGDQ HW��DO@
7VXKDQ�&KHQ
*HQHUDWLRQ�RI�6SULWHImage 1
Image 2
Image N-1
Image N
Image 1
Image 1
Image 2
Image N-1
Image N
Sprite
6WHS����)LQGLQJ�WKH�RIIVHW 6WHS����*HQHUDWLQJ�WKH�VSULWH
7VXKDQ�&KHQ
:HLJKWLQJ� QHHG�WR�ILQG�D�ZHLJKWLQJ�IXQFWLRQ�WR�EOHQG�WKH�
LPDJHV�WR�IRUP�WKH�VSULWH
0
0.001
0.002
0.003
0.004
0.005
0.006
0.007
0.008
0.009
0.01
0 100 200 300 400
Column number
Wei
ght
0
0.001
0.002
0.003
0.004
0.005
0.006
0.007
0.008
0.009
0.01
0 100 200 300 400
Column number
Wei
gh
t
0
0.02
0.04
0.06
0.08
0.1
0 100 200 300 400
Column number
Wei
gh
t
&RQVWDQW�ZHLJKWLQJ 7ULDQJXODU�ZHLJKWLQJ 'HOWD�ZHLJKWLQJ
7VXKDQ�&KHQ
&RQVWDQW�:HLJKWLQJ
««««
� � N�
7VXKDQ�&KHQ
7ULDQJXODU�:HLJKWLQJ
««««
� � N�
7VXKDQ�&KHQ
'HOWD�:HLJKWLQJ
««««
� � N�
7VXKDQ�&KHQ
0RGLILHG�&RGHF
� 3UHGLFWLRQ�IURP�VSULWH�LPDJH�ZLWKRXW�0&
DCT Q
Prediction
IQ+
−� IDCT
Network or Storage
Encoder Decoder
sprite + offset
Prediction
�
sprite + offset
7VXKDQ�&KHQ
:LWK�0RWLRQ�&RPSHQVDWLRQ
� 3UHGLFWLRQ�IURP�VSULWH�LPDJH�ZLWK�0&
DCT Q
Prediction
IQ
ME
MC
MV
+
−� IDCT
Network or Storage
Encoder Decoder
Sprite + offset
Prediction
MC
MV
Sprite + offset
�
7VXKDQ�&KHQ
:LWK�YV��:LWKRXW�0&
� �ZLWKRXW�0& ZLWK�0&
7VXKDQ�&KHQ
7HVW�6HTXHQFHV����
6\QWKHWLF�VHTXHQFH����1HW,&(�URRP 6\QWKHWLF�VHTXHQFH����3DUN
7VXKDQ�&KHQ
7HVW�6HTXHQFHV����
5HDO�VHTXHQFH����.LGV 5HDO�VHTXHQFH����.RQJPLDR
>6KXP��HW�DO@
7VXKDQ�&KHQ
:HLJKWLQJ�IXQFWLRQ�5HVXOWV
6\QWKHWLF�VHTXHQFH�� 6\QWKHWLF�VHTXHQFH��
5HDO�VHTXHQFH�� 5HDO�VHTXHQFH��
25
27
29
31
33
35
37
0 0.05 0.1 0.15 0.2
Bit rate (bpp)
PSN
R (d
B)
25
27
29
31
33
35
37
0 0.05 0.1 0.15 0.2
Bit rate (bpp)
PSN
R (d
B)
29
31
33
35
37
39
41
43
0 0.02 0.04 0.06 0.08 0.1
Bit rate (bpp)
PSN
R (d
B)
34
36
38
40
42
44
0 0.01 0.02 0.03 0.04 0.05
Bit rate (bpp)
PSN
R (d
B)
&RQVWDQW�ZHLJKWLQJ�������� 7ULDQJXODU�ZHLJKWLQJ���� 'HOWD�ZHLJKWLQJ
7VXKDQ�&KHQ
&RPSUHVVLRQ�5HVXOW
6\QWKHWLF�VHTXHQFH�� 6\QWKHWLF�VHTXHQFH��
29
31
33
35
37
39
41
0 0.05
Bit rate (bpp)
PS
NR
(dB
)
Intra coding Inter coding Mosais without MC Mosaic with MC
34
36
38
40
42
44
0 0.01 0.02 0.03 0.04 0.05
Bit rate (bpp)P
SN
R (d
B)
Intra coding Inter coding Mosais without MC Mosaic with MC
,QWUD�FRGLQJ���� ,QWHU�FRGLQJ���������6SULWH�ZLWKRXW�0&��������6SULWH�ZLWK�0&
7VXKDQ�&KHQ
&RPSUHVVLRQ�5HVXOW
5HDO�VHTXHQFH�� 5HDO�VHTXHQFH��
2526272829303132333435
0 0.05 0.1
Bit rate (bpp)
PS
NR
(dB
)
Intra coding Inter coding Mosais without MC Mosaic with MC
25262728293031323334
0 0.05 0.1
Bit rate (bpp)P
SN
R (d
B)
Intra coding Inter coding Mosais without MC Mosaic with MC
,QWUD�FRGLQJ���� ,QWHU�FRGLQJ���������6SULWH�ZLWKRXW�0&��������6SULWH�ZLWK�0&
7VXKDQ�&KHQ
(QKDQFHPHQWV:LQGRZ�VL]H�IRU�VHDUFKLQJ�RIIVHWV
6WULSH�PRWLRQ�FRPSHQVDWLRQ
0&�XVLQJ�D�ODUJH�UHIHUHQFH�IUDPH
0XOWLSOH�VSULWHVKids
26
27
28
29
30
31
32
33
0.2 0.3 0.4 0.5 0.6 0.7 0.8
bit rate (bpp)
PS
NR
(d
B)
1 sprite 3 sprites 5 sprites 7 sprites 9 sprites
7VXKDQ�&KHQ
5HFDS«6SULWH�SUHGLFWLRQ�ZLWK�0&�EHWWHU�WKDQ�,QWUD�FRGLQJ6SULWH�SUHGLFWLRQ�ZLWK�0&�LV�SUHIHUUHG�IRU�UDQGRP�DFFHVV%HWWHU�WKDQ�,QWHU�FRGLQJ�IRU�UHDO�GDWD
'HOWD�ZHLJKWLQJ�LV�WKH�EHVW�IRU�FRQVWUXFWLQJ�WKH�VSULWH&DQ�EH�H[WHQGHG�WR�KLJKHU�GLPHQVLRQV/XPLJUDSK��OLJKWILHOG��HWF�
7VXKDQ�&KHQ
6WUHDPLQJ��'
��VHFRQG
���VHFRQGV���VHFRQGV
*HRPHWU\�7H[WXUH
7VXKDQ�&KHQ
7H[WXUH���*HRPHWU\� ��'�2EMHFW
&RUQHU�%DVHG
9HUWH[�%DVHG
7VXKDQ�&KHQ
:K\�&RPSUHVVLRQ"(DFK�YHUWH[��WKUHH�IORDWLQJ�SRLQW�QXPEHUV,I�HDFK�YHUWH[�VKDUHG�E\���WULDQJOHV��DQG�PD[�QXPEHU�RI�YHUWLFHV�SHU�PRGHO�LV����
⇒ ����ELWV�WULDQJOH�QHHGHG
⇒ ���.%a�0%�IRU�DQ�DYHUDJH�PRGHO���WH[WXUH
triangle
bits
IDvertex
bits
triangle
IDsvertex
vertex
bits
triangle
vertices 10820*
33*32*
3*
6
1 =
+
&RPSUHVVLRQ�RI��'�2EMHFWV7H[WXUH�FRPSUHVVLRQ6WDWLF�WH[WXUHV��-3(*�RU�-3(*�����'\QDPLF�WH[WXUHV��03(*�RU�+����
*HRPHWU\�FRPSUHVVLRQ4XDQWL]DWLRQ�RI�YHUWH[�FRRUGLQDWHV3UHGLFWLYH�FRGLQJ(QWURS\�FRGLQJ
*UDQXODU�VWDEOH�SURJUHVVLYH�FRGLQJ0HVK�RSWLPL]DWLRQ�VLPSOLILFDWLRQ�
>+RSSH�HW�DO@>+HFNEHUW HW�DO@>6FKURGHU HW�DO@>7DXELQ HW�DO�@
7VXKDQ�&KHQ
7H[WXUH�&RGLQJ
%ORFN�'LDJUDP
9HUWH[�4XDQWL]DWLRQ
(QWURS\�&RGLQJ
9HUWH[�&RRUGLQDWHV
3UHGLFWLRQ
%LWVWUHDP
&RQQHFWLYLW\��
�
�'�0RGHO7H[WXUH
7VXKDQ�&KHQ
(QFRGLQJ9HUWH[�GHFLPDWLRQ
C
C
1
234
56
165
4
5H�WULDQJXODWLRQ
( )iv
7VXKDQ�&KHQ
,PSRUWDQFH�RI�9HUWLFHV
��� 9ROXPH
��� &RORU
)(iv
)(ic
V1V2
7VXKDQ�&KHQ
5DQN�DOO�YHUWLFHV�IURP�KLJK�WR�ORZ�EDVHG�RQ�D�FRVW�IXQFWLRQ�
Y��L���LV�WKH�JHRPHWU\�FRVWF��L���LV�WKH�WH[WXUH�FRORU�QRUPDO�FRVWα LV�DQ�XVHU�VSHFLILHG�SDUDPHWHU
'HFLPDWH�WKH�YHUWLFHV�ZLWK�ORZ�FRVW�ILUVW7UDQVPLW�WKH�YHUWLFHV�ZLWK�KLJK�FRVW�ILUVW
)()1()()( icivim αα −+=
7VXKDQ�&KHQ
&RGLQJ�RI�7H[WXUH9HUWH[�EDVHG:DYHOHW��63,+7����HQWURS\�FRGLQJ
&RUQHU�EDVHG3DGGLQJ���'&7���UXQ�OHQJWK�FRGLQJ���HQWURS\�FRGLQJ7H[WXUH�UH�PDSSLQJ�QHHGHG
7VXKDQ�&KHQ
7H[WXUH�5H�0DSSLQJ
m
v
vm
7VXKDQ�&KHQ
���
���
���
��
���
���
��
��
950/
��J]LS
WH[WXUH����9DVH
WH[WXUH���'XFN
QRQH����3LHWD
WH[WXUH����7RWHP
QRQH����+RUVH
QRQH����&URFRGLOH
1RQH����&RZ
1RQH����%HHWKRYHQ
$WWULEXWHV03(*��2XU
$OJRULWKP
�LQ�.%\WHV�
&RPSDULVRQ
7VXKDQ�&KHQ
9LHZ�$GDSWLYH�7UDQVPLVVLRQ
Viewpoint B
Viewpoint A
Hypothetical Viewpoint
7VXKDQ�&KHQ
5HWULHYDO�RI��'�2EMHFWV,QGH[LQJ�DQG�UHWULHYDO0XFK�LV�GRQH�IRU�LPDJHV>+XDQJ�HW�DO@>&R[�HW�DO@
5HFHQW�ZRUN�IRU��'�REMHFWV5HODWHG�WR�03(*��
)HDWXUH�H[WUDFWLRQ
)HDWXUH�PDWFKLQJ
7VXKDQ�&KHQ
)HDWXUH�([WUDFWLRQ)HDWXUH�H[WUDFWLRQ7UDGLWLRQDOO\�YHUWH[�VXUIDFH�EDVHG1HZ�UHJLRQ�EDVHG�IHDWXUHVPRPHQW�LQYDULDQWV��)RXULHU�WUDQVIRUP�FRHIILFLHQWV��HWF��
3UHSURFHVVLQJ�WR�FORVH�WKH�PRGHO
6XUIDFH 5HJLRQ
7VXKDQ�&KHQ
)HDWXUH�([WUDFWLRQ��FRQW��(IILFLHQWO\�FDOFXODWH�UHJLRQ�EDVHG�IHDWXUH�GLUHFWO\�IURP�PHVK6LJQHG�IHDWXUH�IRU�HDFK�PHVK�HOHPHQW5REXVW�WR�WULDQJXODWLRQ$SSOLHV�WR�DQ\�IHDWXUH�WKDW�FDQ�EH�GHFRPSRVHG�WR�HDFK�PHVK�HOHPHQW +
−−
−+
+
7VXKDQ�&KHQ
�'�0RGHO�5HWULHYDO
7VXKDQ�&KHQ
$QQRWDWLRQ�DQG�$FWLYH�/HDUQLQJ
6HPDQWLF�WKUX�DQQRWDWLRQ�LV�QHHGHG/RZ�OHYHO�IHDWXUHV�QRW�HQRXJK+LHUDUFKLFDO�DQQRWDWLRQ&RPSDWLEOH�FRQFHSWV�LQ�DQQRWDWLRQ
$FWLYH�OHDUQLQJ&RPSOHWH�DQQRWDWLRQ�LV�LPSUDFWLFDO6HOHFW�WKH�REMHFW�PRVW�XQFHUWDLQ�IRUDQQQRDWLRQ
7VXKDQ�&KHQ
$QQRWDWLRQ
7VXKDQ�&KHQ
$FWLYH�/HDUQLQJ)RU�HDFK�PRGHO��HDFK�FRQFHSW��ZH�PDLQWDLQ�D�SUREDELOLW\�RI�WKLV�PRGHO�EHORQJLQJ�WR�WKLV�FRQFHSW
6HW�WKH�SUREDELOLW\�WR���RU���LI�DQQRWDWHG
(VWLPDWH�SUREDELOLWLHV�RI�WKH�XQODEHOHG�REMHFWV�ZLWK�SRWHQWLDO�IXQFWLRQ
8VH�WKH�SUREDELOLWLHV�WR�HVWLPDWH�XQFHUWDLQW\�DQG�WR�PHDVXUH�WKH�VHPDQWLF�GLVWDQFH
7VXKDQ�&KHQ
$FWLYH�/HDUQLQJ
-5 -4 -3 -2 -1 0 1 2 3 4 50.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
max2 d⋅
d
p )/exp(5.05.0 2max
20 ddcp ⋅−+=
-10 -8 -6 -4 -2 0 2 4 6 8 100.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
f
kp
Annotated models
One annotated neighborhood Multiple annotated neighborhoods
The potential function
7VXKDQ�&KHQ
(VWLPDWH�WKH�8QFHUWDLQW\
1
p2p
iep
7VXKDQ�&KHQ
5HVXOWV
0
0.5
1
1.5
2
2.5
3
0 50 100 150 200 250 300 350
# of Samples Annotated
Ret
riev
al P
erfo
rman
ce (D
)
Best Gradient Search
Random Sampling
Our Algorithm
0
1
2
3
4
5
6
0 50 100 150 200 250
# of Models Annotated
Ret
riev
al P
erfo
rman
ce (D
) Best Gradient Search
Random Sampling
Our Algorithm
Synthetic database A small database
7VXKDQ�&KHQ
5HVXOWV��FRQW��
0
1
2
3
4
5
6
7
8
9
0 500 1000 1500 2000
# of models annotated
Ret
riev
al P
erfo
rman
ce (
D) Random Sampling
Our algorithm
7VXKDQ�&KHQ
5HFDS«1HZ�IHDWXUH�VHW�IRU��'�PRGHOV
$FWLYH�OHDUQLQJ�WR�LPSURYH�DQQRWDWLRQ�HIILFLHQF\
&RPSDWLEOH�FRQFHSW�WUHH�IRU�DQQRWDWLRQ
3UREDELOLW\�IRU�ERWK�XQFHUWDLQW\�HVWLPDWLRQ�DQG�VHPDQWLF�GLVWDQFH
“Collaboration from anywhere, through any media, as if face-to-face in one room”
Network
,PPHUVLYH�(QYLURQPHQWV
7VXKDQ�&KHQ
7VXKDQ�&KHQ
$�3URWRW\SH1HW,&(��1HWZRUNHG�,QWHOOLJHQW�&ROODERUDWLYH�(QYLURQPHQW/LS�V\QF�IDFLOLWDWHV�VSHHFK�XQGHUVWDQGLQJ
:KR�LV�VSHDNLQJ�DQG�ZKDW�LV�EHLQJ�VDLG
&RQVLVWHQW�VSDWLDO�UHODWLRQVKLS�ZLWK�H\H�FRQWDFW:KRP�LV�VSRNHQ�WR
)DFLDO�H[SUHVVLRQV�DQG�YRLFH�GULYHQ�KDQG�JHVWXUHV'LUHFWLRQDO�VRXQG�JLYH�VHQVH�RI�GLVWDQFH�DQG�GLUHFWLRQ
:KR�LV�ZKHUH��:KR�LV�VSHDNLQJ(QDEOH�VPDOO�JURXS�LQWHUDFWLRQ�LQ�D�URRP�IXOO�RI�SHRSOH
,QIRUPDWLRQ�VKDULQJ6KDUHG�ZKLWHERDUG6WUHDPLQJ��'�REMHFWV(QDEOH�FROODERUDWLYH�GHVLJQ��H�J���FDUV��EXLOGLQJV��HWF�
7VXKDQ�&KHQ
1HW,&(
7VXKDQ�&KHQ
1HW,&(
7VXKDQ�&KHQ
1HW,&(
7VXKDQ�&KHQ
&DVH�6WXG\��2QOLQH�$XFWLRQ
7VXKDQ�&KHQ
2QJRLQJ�:RUN8VH�,%5�IRU�EDFNJURXQG�UHQGHULQJ
8VHU�VWXG\7RJHWKHU�RU�RQ�ORFDWLRQ
7UDFNLQJ�IRU�UHQGHULQJ+HDG�WUDFNLQJ�IRU�KHDG�RULHQWDWLRQ*D]H�WUDFNLQJ�IRU�H\H�FRQWDFW+DQG�WUDFNLQJ�IRU�KDQG�JHVWXUHV
7VXKDQ�&KHQ
6XPPDU\&RPSUHVVLRQ�IRU�,%5
&RPSUHVVLRQ�IRU��'�PHVKHV
,QGH[LQJ�DQG�UHWULHYDO�RI��'�REMHFWV
,PPHUVLYH�HQYLURQPHQWV
7VXKDQ�&KHQ
$GYDQFHG�0XOWLPHGLD�3URFHVVLQJ�/DE
3OHDVH�YLVLW�XV�DW�
KWWS���DPS�HFH�FPX�HGX