[Open.ogc] Deleted layers still exist?
Micah Wengren
micah.wengren at noaa.gov
Wed Sep 25 15:39:51 UTC 2013
Hi John,
I am working on troubleshooting this issue with OpenGeo/Boundless
support. I've gotten the response back that GeoServer should be able to
modify an underlying table schema when a new shapefile is uploaded with
changes in the attribution, but I think based on your evidence more
testing of this is needed.
I asked whether it made a difference if the layer (and associated
featuretype) were currently published when the new data upload happened,
or if they should first be deleted. If you get a chance, could you test
with the same layer you used below 'communities_pgs' by first deleting
the layer via Web UI and then doing the data upload (incl new schema)
with the hpcc_data_requests.py script?
Micah
On 9/24/2013 4:14 PM, John Kennedy - NOAA Affiliate wrote:
> Interesting question. I did a preview and the features do appear to be
> overwritten in the layer. But the "new" attributes are not loading as
> the column names defer between the old and the new shape file. I
> checked a number of polygons and got blank values as shown below.
>
> Inline image 1
>
> John F. Kennedy, PhD, CPG
> GIS Specialist
> Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
> Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
> <mailto:Tim.Haverland at noaa.gov>>
> Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov
> <mailto:hussain.jabalpurwala at noaa.gov>>
>
>
> On Tue, Sep 24, 2013 at 4:05 PM, Micah Wengren <micah.wengren at noaa.gov
> <mailto:micah.wengren at noaa.gov>> wrote:
>
> Sure, no problem. So if you take your new shapefile with modified
> columns and try to upload it to the original layer name
> 'communities_pgs', can you tell for sure that it does not
> overwrite data in the columns that haven't changed from the old
> version to the new? It should be doing that at least, and
> discarding any new columns, as well as probably leaving any
> original columns that were removed in the new file empty. At
> least that would be better than not overwriting anything at all.
>
> Micah
>
>
>
> On 9/24/2013 3:47 PM, John Kennedy - NOAA Affiliate wrote:
>> If the name of the shape file is the same as one of the layers,
>> then geoserver presents the old data. I took the shape file I'm
>> attempting to load as communities_pgs and named it
>> communities_pgs20130924 and it loads as expected with the new
>> columns and data.
>>
>> Thanks for all your help on this issue.
>>
>> John F. Kennedy, PhD, CPG
>> GIS Specialist
>> Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149 <tel:301-427-8149>
>>
>> Comments? Have I been of assistance? Please let my supervisors
>> know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>> <mailto:Tim.Haverland at noaa.gov>>
>> Hussain Jabalpurwala - NOAA Affiliate
>> <hussain.jabalpurwala at noaa.gov
>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>
>>
>> On Tue, Sep 24, 2013 at 3:32 PM, Micah Wengren
>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>
>> John,
>>
>> You may have hit on an issue with the way our system works.
>> Can you tell if the attribute values that are being saved in
>> the database are coming from the new shapefile you just tried
>> to upload? Could you insert a test value and try the upload
>> again? I just want to check that the new data is actually
>> being written when you specify update=overwrite. It should
>> be, but it would be good to confirm.
>>
>> I don't actually know how GeoServer handles the overwrite
>> option behind the scenes. I thought it might drop the
>> underlying database table and recreate, but it may not be.
>> That would be why the schema is being preserved. Best way to
>> find out is for me to open a ticket with OpenGeo. I'll do
>> that later today.
>>
>> Micah
>>
>>
>> On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
>>> Okay, I now feel a little silly. I did not replace the dummy
>>> password in my notes file with my LDAP password. Shape files
>>> and zip files have the same names as the layer. I attempted
>>> the data load again, using --update overwrite, and I got a
>>> 201 response. However, when I look at the layer in geoserver
>>> I see the old data. There is a column (property) called
>>> GEOID_jlf and that column is in the old data layer and not
>>> in the new layer I'm attempting to load.
>>>
>>> Inline image 1
>>>
>>>
>>> John F. Kennedy, PhD, CPG
>>> GIS Specialist
>>> Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149 <tel:301-427-8149>
>>>
>>> Comments? Have I been of assistance? Please let my
>>> supervisors know. :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>> <mailto:Tim.Haverland at noaa.gov>>
>>> Hussain Jabalpurwala - NOAA Affiliate
>>> <hussain.jabalpurwala at noaa.gov
>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren
>>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>>
>>> Ok. Just curious, what are the shapefile names you're
>>> trying to upload? Are they the communities_pts and
>>> communities_pgs layers?
>>>
>>> Looks like those are gone from the FeatureTypes list now
>>> (https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
>>> But they still show up in the list when you go to the
>>> 'New Layer' page. Not sure I've seen that issue before,
>>> so might have to file a ticket.
>>>
>>> As for credentials, it should only be your NOAA LDAP
>>> username and password. Those don't get passed on to the
>>> database, there's a separate way database authorization
>>> is handled. If you can log in to the Web UI, there
>>> shouldn't be an issue with the REST API, or at least you
>>> shouldn't be getting that 401 response.
>>>
>>> Micah
>>>
>>>
>>> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate wrote:
>>>> Thanks. Tomorrow works better for me. As for the
>>>> credentials, nothing as changed on this end. I was able
>>>> to load data with the hpcc_data.py script up to
>>>> yesterday, but I did get an error message when
>>>> attempting to overwrite the layer/feature type. Now,
>>>> there maybe credential issues in the database
>>>> (PostGIS?) but I've no way of knowing.
>>>>
>>>> John F. Kennedy, PhD, CPG
>>>> GIS Specialist
>>>> Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149 <tel:301-427-8149>
>>>>
>>>> Comments? Have I been of assistance? Please let my
>>>> supervisors know. :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>> <hussain.jabalpurwala at noaa.gov
>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren
>>>> <micah.wengren at noaa.gov
>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>
>>>> Hi John,
>>>>
>>>> That is odd for the 401 response on the data
>>>> upload. Are you sure the credentials are right?
>>>> The other two seem ok, depending on if the
>>>> featuretype existed in the first place or not.
>>>>
>>>> I might be able to come by later, but right now I
>>>> have a deadline to finish something this afternoon.
>>>> Let me know if you aren't able to get it to work.
>>>>
>>>> Micah
>>>>
>>>>
>>>> On 9/24/2013 12:32 PM, John Kennedy - NOAA
>>>> Affiliate wrote:
>>>>> Hi Micah,
>>>>>
>>>>> Well, I think I'm getting close. I installed pip,
>>>>> python setup, and request. I used
>>>>> hpcc_date_requests.py to delete the layer and the
>>>>> feature type and I get these to responses:
>>>>> Response: 200,
>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>
>>>>> Response: 404,
>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>
>>>>> The hpcc_date_requests appears to be attempting to
>>>>> load data, but the response back is this:
>>>>> Response: 401,
>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>>>
>>>>> Any chance you can come to my cube and have a look?
>>>>>
>>>>> Cheers,
>>>>> John
>>>>>
>>>>>
>>>>>
>>>>> John F. Kennedy, PhD, CPG
>>>>> GIS Specialist
>>>>> Contractor: Caelum Corp.
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 9518
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8149 <tel:301-427-8149>
>>>>>
>>>>> Comments? Have I been of assistance? Please let my
>>>>> supervisors know. :~)
>>>>> Tim Haverland - NOAA Federal
>>>>> <Tim.Haverland at noaa.gov
>>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>>> <hussain.jabalpurwala at noaa.gov
>>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>>
>>>>>
>>>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah Wengren
>>>>> <micah.wengren at noaa.gov
>>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>>
>>>>> Hi John,
>>>>>
>>>>> How are you doing the attempted data upload?
>>>>> With the python script? Can you tell me which
>>>>> python script you are using if so? If you can,
>>>>> try using the hpcc_data_requests.py script
>>>>> instead of the other one (might involve some
>>>>> admin rights to install python modules on your
>>>>> workstation). I've update that one more
>>>>> recently. I'd like to drop the other one at
>>>>> some point for easier maintenance.
>>>>>
>>>>> I think you should be able to upload a new
>>>>> data set to replace the existing one, even if
>>>>> it already exists in the database and data
>>>>> store (but could be wrong if there's an issue
>>>>> going on here with these layers). Something
>>>>> like the following should work:
>>>>>
>>>>> > hpcc_data_requests.py -a data -u <name> -p
>>>>> <pass> -w nmfs_st -f /path/to/shapefile.zip
>>>>> --type shape --update overwrite (the shapefile
>>>>> would need to be named to match the layer you
>>>>> want to update).
>>>>>
>>>>>
>>>>> Also, this is something that is a little
>>>>> whacky with the GeoServer REST API (and should
>>>>> be better documented somewhere admittedly),
>>>>> but you may need to do a delete command for
>>>>> both the Layer and the FeatureType to fully
>>>>> delete it. When you create a new layer with
>>>>> the Web UI, it actually makes both the Layer
>>>>> and FeatureType behind the scenes. Not sure
>>>>> if this will fix this particular issue, but
>>>>> commands to do that would be something like:
>>>>>
>>>>> > hpcc_data_requests.py -a delete_layer -u
>>>>> <name> -p <pass> -w nmfs_st --layername
>>>>> <layername>
>>>>> > hpcc_data_requests.py -a delete_featuretype
>>>>> -u <name> -p <pass> -w nmfs_st --layername
>>>>> <layername>
>>>>>
>>>>> This should fully delete both a layer and its
>>>>> corresponding featuretype. It's possible if
>>>>> you did a delete_layer command with via the
>>>>> script without doing delete_featuretype as
>>>>> well, some featuretypes got left behind.
>>>>> Again, an issue that should really be better
>>>>> documented in GeoServer docs but unfortunately
>>>>> I haven't seen it described anywhere, I
>>>>> happened to find out myself after having some
>>>>> similar issues.
>>>>>
>>>>> I'm looking around the workspace now, and I
>>>>> think I see some cases where there are
>>>>> FeatureTypes that aren't associated with any
>>>>> layers, in particular 'communities_pgs' and
>>>>> 'communities_pts' are both FeatureTypes
>>>>> without corresponding layers. This could be
>>>>> preventing the data upload from happening for
>>>>> these layers. I'm just looking at these two
>>>>> lists:
>>>>>
>>>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage
>>>>> (filtered by 'nmfs_st')
>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>>>
>>>>> Also, you should probably do the
>>>>> delete_featuretype command for
>>>>> 'communities_pgs20130506' and
>>>>> 'communities_pts20130506', since they don't
>>>>> have corresponding Layers. I think that will
>>>>> get those two in sync.
>>>>>
>>>>> I'm not sure why you get such a long list in
>>>>> the New Layer dialog though, looks like a bug
>>>>> to me. No idea where
>>>>> 'SI_SE_Gulf_sites_pg_ALLsites' and
>>>>> 'SI_SE_Gulf_sites_pt' are coming from, because
>>>>> those aren't listed as current FeatureTypes in
>>>>> the REST API.
>>>>>
>>>>> HTH,
>>>>> Micah
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA
>>>>> Affiliate wrote:
>>>>>> The two layers of concern are communities_pgs
>>>>>> and communities_pts. When I attempt to load
>>>>>> new data, via a zip file with a shape file, I
>>>>>> get a message that the communities_pgs
>>>>>> exists. When I look at the list of source
>>>>>> under the "Add sources" button I can see
>>>>>> the communities_pgs layer and the publish
>>>>>> link. But when I click the link and attempt
>>>>>> to save I get the message below. Deleting
>>>>>> layer via the hpcc_data python script also
>>>>>> seems to fail.
>>>>>>
>>>>>> Inline image 1
>>>>>>
>>>>>> John F. Kennedy, PhD, CPG
>>>>>> GIS Specialist
>>>>>> Contractor: Caelum Corp.
>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 9518
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8149 <tel:301-427-8149>
>>>>>>
>>>>>> Comments? Have I been of assistance? Please
>>>>>> let my supervisors know. :~)
>>>>>> Tim Haverland - NOAA Federal
>>>>>> <Tim.Haverland at noaa.gov
>>>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>>>> <hussain.jabalpurwala at noaa.gov
>>>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim
>>>>>> Haverland - NOAA Federal
>>>>>> <tim.haverland at noaa.gov
>>>>>> <mailto:tim.haverland at noaa.gov>> wrote:
>>>>>>
>>>>>> I'm adding John Kennedy to this thread so
>>>>>> he can work with you on this issue.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 24, 2013 at 8:49 AM, Micah
>>>>>> Wengren <micah.wengren at noaa.gov
>>>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>>>
>>>>>> Tim,
>>>>>>
>>>>>> I can see some layers in your
>>>>>> workspace in the New Layer dialog
>>>>>> that aren't currently published and
>>>>>> have a 'Publish' link next to them,
>>>>>> so I see what you mean. Are you able
>>>>>> to create a new SQL view layer with
>>>>>> the same name as one of those layers,
>>>>>> or does that not work? Can you give
>>>>>> me the name of one of the layers that
>>>>>> you've tried with and I'll pass it on
>>>>>> to OpenGeo support to troubleshoot,
>>>>>> that way I can give them a specific
>>>>>> layer name with the issue.
>>>>>>
>>>>>> The layers should be disappearing but
>>>>>> for whatever reason they're not.
>>>>>>
>>>>>> Micah
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 9/23/2013 3:55 PM, Tim Haverland -
>>>>>> NOAA Federal wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We've been trying to clean up a
>>>>>>> bunch of layers in our workspace. We
>>>>>>> can delete layers using the "Remove
>>>>>>> selected resources" button on the
>>>>>>> Layers page, but when we try to
>>>>>>> replace that layer with another, the
>>>>>>> layer is still there.
>>>>>>>
>>>>>>> We can verify this by clicking "Add
>>>>>>> a new resource" ... all of the old
>>>>>>> layers are still there and I can
>>>>>>> click Publish to make them "live" again.
>>>>>>>
>>>>>>> How do we permanently get rid of
>>>>>>> unwanted layers?
>>>>>>>
>>>>>>> Tim
>>>>>>> --
>>>>>>> *Tim Haverland*
>>>>>>> Acting Operations Branch Chief
>>>>>>> NOAA Fisheries Office of Science and
>>>>>>> Technology
>>>>>>> 1315 East-West Highway
>>>>>>> SSMC3 Rm 12303
>>>>>>> Silver Spring, MD 20910
>>>>>>> 301-427-8137 <tel:301-427-8137>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Open.ogc mailing list
>>>>>>> Open.ogc at list.woc.noaa.gov <mailto:Open.ogc at list.woc.noaa.gov>
>>>>>>> https://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Tim Haverland*
>>>>>> Acting Operations Branch Chief
>>>>>> NOAA Fisheries Office of Science and
>>>>>> Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 12303
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8137 <tel:301-427-8137>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/b62126e5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166250 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/b62126e5/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/b62126e5/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/b62126e5/attachment-0005.png>
More information about the Open.ogc
mailing list